[Azure Logs] Add dedicated aadgraphactivitylogs data stream by terrancedejesus · Pull Request #18880 · elastic/integrations

terrancedejesus · 2026-05-07T18:31:06Z

Adds the azure.aadgraphactivitylogs data stream to ingest the AzureADGraphActivityLogs diagnostic category from Microsoft Entra ID, parallel to azure.graphactivitylogs for Microsoft Graph. Without this, AAD Graph events fall through to azure.platformlogs and the AAD-Graph-specific properties survive only inside event.original.

Proposed commit message

azure: add aadgraphactivitylogs data stream

Add a dedicated data stream for the AzureADGraphActivityLogs
diagnostic category from Microsoft Entra ID. Without this,
legacy Azure AD Graph (graph.windows.net) events fall through
to the platformlogs catch-all and lose schema-aware parsing.

The events router maps routing.category ==
"AzureADGraphActivityLogs" to the new dataset. The ingest
pipeline extracts ECS fields: event.action from HTTP method +
URI collection, event.outcome from response status,
event.category [iam, web], and related.user including the
OAuth app_id for client correlation.

Legacy AAD Graph is still actively used by Microsoft first-party
tooling, older line-of-business apps, and adversary tooling
(ROADtools, AzureHound v1, AADInternals). The dedicated dataset
makes these events available for detection rules and dashboards.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices — N/A, no dashboards added.

Author's Checklist

PII redacted from pipeline test fixtures (tenant IDs, user/session/request UUIDs, sign-in activity IDs, internal user-agents). Synthetic test values throughout.
elastic-package check passes; elastic-package test pipeline -d aadgraphactivitylogs passes.
End-to-end stack test: events POSTed to logs-azure.events-default are correctly rerouted to logs-azure.aadgraphactivitylogs-default with full ECS field extraction.

How to test this PR locally

cd packages/azure
elastic-package check
elastic-package stack up -d -v
elastic-package install
elastic-package test pipeline -d aadgraphactivitylogs

Optional end-to-end:

# Pipe the captured fixtures through the live events router
python3 -c "
import json
with open('data_stream/aadgraphactivitylogs/_dev/test/pipeline/test-aadgraph-activity.log') as f:
    for line in f:
        line = line.strip()
        if line:
            print(json.dumps({'create': {'_index': 'logs-azure.events-default'}}))
            print(json.dumps({'message': line}))
" > /tmp/aadgraph-bulk.ndjson

curl -sk -u user:pass -X POST \
  "https://localhost:9200/_bulk?refresh=wait_for" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @/tmp/aadgraph-bulk.ndjson

curl -sk -u user:pass  \
  "https://localhost:9200/logs-azure.aadgraphactivitylogs-default/_count"
# expected: {"count":4,...}

Open Discover at https://localhost:5601 (user / pass)
data view logs-azure.aadgraphactivitylogs-*
confirm event.action, event.outcome, http.*, url.path, azure.aadgraphactivitylogs.properties.*, and related.user all populate.

Related issues

Closes [Azure]: Extend Support to AAD Graph Activity Logs #18879

Screenshots

Pipeline tests passing locally

Discover view of the new dataset with ECS-parsed events
Expanded document showing the full ECS field tree

Adds the azure.aadgraphactivitylogs data stream to ingest the AzureADGraphActivityLogs diagnostic category from Microsoft Entra ID, parallel to azure.graphactivitylogs for Microsoft Graph. Without this, AAD Graph events fall through to azure.platformlogs and the AAD-Graph-specific properties survive only inside event.original.

efd6

Please update the proposed commit message so that it's something that can be used in context of git; no Markdown, appropriately wrapped etc.

For example based on the current code here (update as needed):

azure: add aadgraphactivitylogs data stream

Add a dedicated data stream for the AzureADGraphActivityLogs
diagnostic category from Microsoft Entra ID. Without this,
legacy Azure AD Graph (graph.windows.net) events fall through
to the platformlogs catch-all and lose schema-aware parsing.

The events router maps routing.category ==
"AzureADGraphActivityLogs" to the new dataset. The ingest
pipeline extracts ECS fields: event.action from HTTP method +
URI collection, event.outcome from response status,
event.category [iam, web], and related.user including the
OAuth app_id for client correlation.

Legacy AAD Graph is still actively used by Microsoft first-party
tooling, older line-of-business apps, and adversary tooling
(ROADtools, AzureHound v1, AADInternals). The dedicated dataset
makes these events available for detection rules and dashboards.

terrancedejesus · 2026-05-08T03:20:14Z

@efd6 proposed commit message updated. Thank you!
Any specific labeling or additional checks?

efd6 · 2026-05-08T03:30:04Z

The build is complaining:

Error: error validating packages in directory 'packages': error checking data streams from 'packages/azure': package "packages/azure" shares ownership across data streams but these ones [packages/azure/data_stream/aadgraphactivitylogs] lack owners

I think you will need to add a line before this. Who will be the owner of this data stream?

terrancedejesus · 2026-05-08T03:59:36Z

Who will be the owner of this data stream?

Yes, I noticed the buildkite fail related to owners. There are a few owners across the Azure package data streams it seems so I am not sure what team should be the owner/maintainer? I assume since these are the legacy data stream for Microsoft Graph and it was provisioned to write threat detection rules on, we mirror that so @elastic/security-service-integrations?

elastic-vault-github-plugin-prod · 2026-05-08T04:48:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

chrisberkhout · 2026-05-11T08:53:50Z

This pipeline does match about half the other pipelines with the same name:

find -name 'azure-shared-pipeline.yml' | xargs md5sum | sort

28624170d9ba87d593c9aef7dd72284a ./data_stream/application_gateway/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml 28624170d9ba87d593c9aef7dd72284a ./data_stream/firewall_logs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml 352b4a0232fcf818b45a174958b161e8 ./data_stream/identity_protection/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml 352b4a0232fcf818b45a174958b161e8 ./data_stream/provisioning/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml 8fb30aa0822189b990f17ba026aeb928 ./data_stream/platformlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/activitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/auditlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/eventhub/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/graphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/signinlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml d9d42c87fb050f4264f140fe5c4fddd0 ./data_stream/springcloudlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml

It's a good time to use the new links functionality, at least for the ones that do still match.

Added per recommendations: 7be1278fcf4bcd9e2723782c95c43f5a55f45a2d

Note, I did not adjust all of them that could be as that seems out of scope for this. Maybe a separate issue/pr for this?

It's a step in the right direction. I think it would be good to move that version of the file to the top level and change the identical copies to links. Could be done in this PR.

# place the shared copy at the top level mkdir -p ./_dev/shared/ mv ./data_stream/graphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml ./_dev/shared/ # update link to the shared copy echo "../../../../_dev/shared/azure-shared-pipeline.yml d9410a8b01f785a2e328560cc5d9a2286c8ca5b4e6ab5ec0edfaba385ddb0fb9" > \ ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link # remove other hard copies rm ./data_stream/activitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml rm ./data_stream/auditlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml rm ./data_stream/eventhub/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml rm ./data_stream/signinlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml rm ./data_stream/springcloudlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml # add other links to the shared copy cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/activitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/auditlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/eventhub/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/graphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/signinlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link cp ./data_stream/aadgraphactivitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link ./data_stream/springcloudlogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml.link

github-actions · 2026-05-11T14:26:36Z

TL;DR

Buildkite failed before tests ran because the PR checkout hook hit a GitHub remote 500 while fetching the target branch, so this is an infrastructure/transient fetch failure rather than a code regression in this PR. Re-run the build first; if it repeats, add retry/backoff around the target-branch git fetch in the hook.

Remediation

Re-run Buildkite build #42650 (same commit 30046ce36184e5916a2b5f69455b6c5b0fe976f6).
If this recurs, harden .buildkite/hooks/post-checkout by retrying git fetch -v origin "${target_branch}" (line 28) with bounded retries/backoff, then re-run Get reference from target branch.

Investigation details

Root Cause

The failing step is Get reference from target branch, which executes the repository post-checkout hook. The hook fetches the PR base branch in .buildkite/hooks/post-checkout and exited non-zero when GitHub returned HTTP 500 during git fetch.

Relevant code path:

.buildkite/hooks/post-checkout:28 → git fetch -v origin "${target_branch}"
.buildkite/hooks/post-checkout:70 invokes checkout_merge, so this fetch is required for all PR builds.

The PR changes are focused on Azure package files/CODEOWNERS and do not modify Buildkite hooks, which supports this being infra/transient rather than a PR logic/config bug.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/42650
Job/step: Get reference from target branch
Key log excerpt:

remote: Internal Server Error
fatal: unable to access 'https://github.com/elastic/integrations.git/': The requested URL returned error: 500
🚨 Error: running "repository post-checkout" shell hook: The repository post-checkout hook exited with status 128

Verification

Not run locally (failure is in CI checkout/bootstrap phase before package checks/tests).

Follow-up

If the retry succeeds, no PR code change is needed. If repeated 500s continue across builds, treat as persistent CI infrastructure issue and add fetch retry logic in the hook.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…vitylogs

github-actions · 2026-05-11T15:19:05Z

✅ Vale Linting Results

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

…nd field schemas

…url.extension

…e as comment in ingest pipeline

…after Microsoft Graph Activity Logs in the supported-services list.

terrancedejesus · 2026-05-11T16:46:52Z

@chrisberkhout - Thank you again for the feedback and guidance. All comments have been addressed. For each comment, I added the commit for easier reference. The latest buildkit build has succeeded.

Additionally, I've rebuilt locally a new stack and ingested AAD Graph Activity Logs to verify logs are as expected and integration updates are accurate. Please let me know if if there is anything else to add/adjust.

zmoog

LGTM, just a few nits.

zmoog · 2026-05-12T08:34:56Z

+      tag: pipeline-azure-shared-pipeline
+  - fingerprint:
+      fields: 
+        - azure.aadgraphactivitylogs.properties.request_uri


The pipeline removes the azure.aadgraphactivitylogs.properties.request_uri field a few lines before if url.original is not null. Should we keep it or replace this with a different field?

@zmoog Good catch and thank you for all the review! If I understand correctly, the remove a few lines up was stripping properties.request_uri before the fingerprint got to use it, so that input was silently dropping out (ignore_missing: true masked it). The hash was effectively only using http.request.id, azure.tenant_id, and properties.time_generated and confirmed this against the data.

Swapped it to use url.original instead, which holds the same value (the uri_parts step keeps a copy via keep_original: true) and is still around at fingerprint time. Re-streamed live events through twice end-to-end and confirmed the _ids are stable now.

fix ref: 8fae9c52502f6fa6d3153d78ea16fa265dc37851

…ingest_pipeline/default.yml Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

muthu-mps

code owner approval!

chrisberkhout

Thanks. Looks good.

I gave a script to adopt links for the other uses of the same version of the ingest pipeline.

elasticmachine · 2026-05-14T13:41:04Z

💚 Build Succeeded

Buildkite Build
Commit: 9ad927a

History

💚 Build #42804 succeeded bbc07f7
💚 Build #42788 succeeded f995b6f
💚 Build #42725 succeeded f8eb858
💚 Build #42717 succeeded 08110a7
💚 Build #42661 succeeded c6ba91a
💔 Build #42660 failed f7a11a1

cc @terrancedejesus

elastic-vault-github-plugin-prod · 2026-05-14T14:23:35Z

Package azure - 1.37.0 containing this change is available at https://epr.elastic.co/package/azure/1.37.0/

terrancedejesus and others added 2 commits May 7, 2026 14:23

Update changelog for azure.aadgraphactivitylogs dataset

2c88052

terrancedejesus commented May 7, 2026

View reviewed changes

Comment thread packages/azure/changelog.yml Outdated

terrancedejesus commented May 7, 2026

View reviewed changes

Comment thread packages/azure/manifest.yml Outdated

terrancedejesus added 2 commits May 7, 2026 14:33

Apply suggestion from @terrancedejesus

7f60d1d

Apply suggestion from @terrancedejesus

c1a75c0

This comment has been minimized.

Sign in to view

terrancedejesus requested a review from chrisberkhout May 7, 2026 19:43

efd6 reviewed May 7, 2026

View reviewed changes

terrancedejesus and others added 3 commits May 8, 2026 00:18

azure/aadgraphactivitylogs: update codeowners

82dd31b

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

9c4fb1e

azure/aadgraphactivitylogs: update codeowners

357c09f

This comment has been minimized.

Sign in to view

terrancedejesus added the Integration:azure Azure Logs label May 8, 2026

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

ecae992

chrisberkhout requested changes May 11, 2026

View reviewed changes

azure/aadgraphactivitylogs: remove log YAML HBS file

30046ce

terrancedejesus added 2 commits May 11, 2026 11:17

azure/aadgraphactivitylogs: adding docs and updated README

0a4f697

azure/aadgraphactivitylogs: adding separate manifest for aadgraphacti…

20ba136

…vitylogs

terrancedejesus added 5 commits May 11, 2026 11:20

azure/aadgraphactivitylogs: adjust data stream test pipeline events a…

8e1dc86

…nd field schemas

azure/aadgraphactivitylogs: fix vale linting warnings

97285cc

azure/aadgraphactivitylogs: adds shared pipeline link

7be1278

azure/aadgraphactivitylogs: remove 8.14 workaround for dynamic field …

a62a1ac

…url.extension

azure/aadgraphactivitylogs: adds aadgraphactivitylogs schema referenc…

99758fd

…e as comment in ingest pipeline

terrancedejesus and others added 2 commits May 11, 2026 11:47

azure/aadgraphactivitylogs: added Azure AD Graph Activity Logs right …

7f1b4b6

…after Microsoft Graph Activity Logs in the supported-services list.

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

f7a11a1

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label May 11, 2026

azure/aadgraphactivitylogs: sample event and documentation fix

c6ba91a

terrancedejesus marked this pull request as ready for review May 11, 2026 16:47

terrancedejesus requested review from a team as code owners May 11, 2026 16:47

terrancedejesus self-assigned this May 11, 2026

andrewkroh added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label May 11, 2026

zmoog approved these changes May 12, 2026

View reviewed changes

terrancedejesus and others added 4 commits May 12, 2026 08:46

Update packages/azure/data_stream/aadgraphactivitylogs/elasticsearch/…

6575463

…ingest_pipeline/default.yml Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

Update packages/azure/data_stream/aadgraphactivitylogs/elasticsearch/…

08110a7

…ingest_pipeline/default.yml Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

azure/aadgraphactivitylogs: url.original keep original from fingerprint

8fae9c5

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

f8eb858

terrancedejesus requested a review from chrisberkhout May 12, 2026 14:56

muthu-mps approved these changes May 13, 2026

View reviewed changes

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

f995b6f

chrisberkhout approved these changes May 13, 2026

View reviewed changes

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

bbc07f7

terrancedejesus requested a review from efd6 May 13, 2026 19:12

Merge branch 'main' into enhancement/azure-ad-graph-activitylogs

9ad927a

terrancedejesus merged commit ce28db2 into main May 14, 2026
12 checks passed

terrancedejesus deleted the enhancement/azure-ad-graph-activitylogs branch May 14, 2026 14:03

Conversation

terrancedejesus commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

terrancedejesus commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented May 8, 2026

Uh oh!

terrancedejesus commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

chrisberkhout May 11, 2026

Choose a reason for hiding this comment

Uh oh!

terrancedejesus May 11, 2026

Choose a reason for hiding this comment

Uh oh!

chrisberkhout May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 11, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

github-actions Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Vale Linting Results

Uh oh!

terrancedejesus commented May 11, 2026

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zmoog May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terrancedejesus May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 14, 2026

💚 Build Succeeded

History

Uh oh!

terrancedejesus commented May 7, 2026 •

edited

Loading

terrancedejesus commented May 8, 2026 •

edited

Loading

terrancedejesus commented May 8, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented May 8, 2026 •

edited

Loading

github-actions Bot commented May 11, 2026 •

edited

Loading

zmoog May 12, 2026 •

edited

Loading

terrancedejesus May 12, 2026 •

edited

Loading